Kubeflow 單節點及多節點安裝重安裝

tags: 研究室碩一筆記

參考連結 : https://kserve.github.io/website/0.8/admin/serverless/#recommended-version-matrix
github : https://github.com/kubeflow/manifests/tree/v1.6.1

單節點安裝

前置作業，記得先

sudo rm -rf /mnt
然後再 
sudo mkdir -p /mnt/minio (因為其他資料夾會自己建立)
並且確定 PVC / PV 沒有多餘的。

1. Install Istio
./kustomize-v3.2.0 build common/istio-1-14/istio-crds/base | kubectl apply -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-namespace/base | kubectl apply -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-install/base | kubectl apply -f -

2. Install Knative Serving
./kustomize-v3.2.0 build common/knative/knative-serving/overlays/gateways | kubectl apply -f -
./kustomize-v3.2.0 build common/istio-1-14/cluster-local-gateway/base | kubectl apply -f -

3. Install Cert Manager
./kustomize-v3.2.0 build common/cert-manager/cert-manager/base | kubectl apply -f -
./kustomize-v3.2.0 build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -

4. Install KServe
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve.yaml

5. Install KServe Built-in ClusterServingRuntimes
kubectl apply -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve-runtimes.yaml

6. Install Other
* Dex
./kustomize-v3.2.0 build common/dex/overlays/istio | kubectl apply -f -

* OIDC AuthService
./kustomize-v3.2.0 build common/oidc-authservice/base | kubectl apply -f -
kubectl apply -f kubeflow-pv.yaml -l name=authservice

* Kubeflow Namespace
./kustomize-v3.2.0 build common/kubeflow-namespace/base | kubectl apply -f -

* Kubeflow Roles
./kustomize-v3.2.0 build common/kubeflow-roles/base | kubectl apply -f -

* Kubeflow Istio Resources
./kustomize-v3.2.0 build common/istio-1-14/kubeflow-istio-resources/base | kubectl apply -f -

* Kubeflow Pipelines
./kustomize-v3.2.0 build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl apply -f -
kubectl apply -f kubeflow-pv.yaml -l name=kubeflow

* katib
./kustomize-v3.2.0 build apps/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -
kubectl apply -f kubeflow-pv.yaml -l name=katib-mysql

* Central Dashboard
./kustomize-v3.2.0 build apps/centraldashboard/upstream/overlays/kserve | kubectl apply -f -

* Admission Webhook
./kustomize-v3.2.0 build apps/admission-webhook/upstream/overlays/cert-manager | kubectl apply -f -

* Notebooks
./kustomize-v3.2.0 build apps/jupyter/notebook-controller/upstream/overlays/kubeflow | kubectl apply -f -

* Profiles + KFAM
./kustomize-v3.2.0 build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -

* Volumes Web App
./kustomize-v3.2.0 build apps/volumes-web-app/upstream/overlays/istio | kubectl apply -f -

* Tensorboard
./kustomize-v3.2.0 build apps/tensorboard/tensorboards-web-app/upstream/overlays/istio | kubectl apply -f -

* Training Operator
./kustomize-v3.2.0 build apps/training-operator/upstream/overlays/kubeflow | kubectl apply -f -

* User Namespace
./kustomize-v3.2.0 build common/user-namespace/base | kubectl apply -f -

解除安裝

./kustomize-v3.2.0 build common/cert-manager/cert-manager/base | kubectl delete -f -
./kustomize-v3.2.0 build common/cert-manager/kubeflow-issuer/base | kubectl delete -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-crds/base | kubectl delete -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-namespace/base | kubectl delete -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-install/base | kubectl delete -f -
./kustomize-v3.2.0 build common/dex/overlays/istio | kubectl delete -f -
./kustomize-v3.2.0 build common/oidc-authservice/base | kubectl delete -f -
./kustomize-v3.2.0 build common/knative/knative-serving/overlays/gateways | kubectl delete -f -
./kustomize-v3.2.0 build common/istio-1-14/cluster-local-gateway/base | kubectl delete -f -
./kustomize-v3.2.0 build common/knative/knative-eventing/base | kubectl delete -f -
./kustomize-v3.2.0 build common/kubeflow-namespace/base | kubectl delete -f -
./kustomize-v3.2.0 build common/kubeflow-roles/base | kubectl delete -f -
./kustomize-v3.2.0 build common/istio-1-14/kubeflow-istio-resources/base | kubectl delete -f -
./kustomize-v3.2.0 build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user | kubectl delete -f -
./kustomize-v3.2.0 build apps/pipeline/upstream/env/platform-agnostic-multi-user-pns | kubectl delete -f -
./kustomize-v3.2.0 build contrib/kserve/kserve | kubectl delete -f -
./kustomize-v3.2.0 build contrib/kserve/models-web-app/overlays/kubeflow | kubectl delete -f -
./kustomize-v3.2.0 build apps/katib/upstream/installs/katib-with-kubeflow | kubectl delete -f -
kubectl delete -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve.yaml
kubectl delete -f https://github.com/kserve/kserve/releases/download/v0.8.0/kserve-runtimes.yaml

Pod 乾淨應該長這樣，也要注意 PV / PVC 都有刪乾淨。

多節點安裝

前置作業，記得先做以下操作，跟單節點不同之處是，
必須先在 MasterNode 將全部安裝完畢之後，才加入 Workernode，

sudo rm -rf /mnt
然後再 
sudo mkdir -p /mnt/minio (因為其他資料夾會自己建立)
並且確定 PVC / PV 沒有多餘的。

1. Install Istio
./kustomize-v3.2.0 build common/istio-1-14/istio-crds/base | kubectl apply -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-namespace/base | kubectl apply -f -
./kustomize-v3.2.0 build common/istio-1-14/istio-install/base | kubectl apply -f -

需特別注意，在此處會遇到 Bug ，原因是因為出於安全考量 Pod 不會被配置到 MasterNode 上，
所以需要執行以下指令，

kubectl taint nodes --all node-role.kubernetes.io/master-

其他安裝指令就跟單節點安裝方式一模一樣了，參照上面的單節點安裝方式即可。

Kubeflow-pv

kind: PersistentVolume
apiVersion: v1
metadata:
  name: katib-mysql-pv
  namespace: kubeflow
  labels:
    type: local
    name: katib-mysql
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/katib-mysql"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: minio-pv
  namespace: kubeflow
  labels:
    type: local
    name: kubeflow
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/minio"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: mysql-pv
  namespace: kubeflow
  labels:
    type: local
    name: kubeflow
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/mysql"
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: authservice-pv
  namespace: istio-system
  labels:
    type: local
    name: authservice
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/autoservice"